Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sharing saved objects phase 3 #94383

Merged
merged 35 commits into from
May 14, 2021

Conversation

jportner
Copy link
Contributor

@jportner jportner commented Mar 10, 2021

Resolves #67380

Changes

This PR primarily makes changes to the server-side SavedObjectsRepository (SOR) and SavedObjectsClient (SOC) methods.

1. Added new SOR/SOC methods:

  • collectMultiNamespaceReferences -- Given an list of object types/IDs, this will collect all outbound multi-namespace references (including transitive references) and return them. It also checks for legacy URL aliases in other spaces and includes that information in the result.
    • Public API route: /api/spaces/_get_shareable_references
    • Note: this is not used by any consumers yet, it will be in a follow-on PR.
  • updateObjectsSpaces -- Given a list of object types/IDs, spaces to add, and spaces to remove, this will update the spaces for each object.
    • Public API route: /api/spaces/_update_objects_spaces

I implemented a new approach for saved objects authorization in these methods. First we fetch the object(s) in question, then we check privileges for all spaces (including any spaces those objects currently exist in). This has the benefit of reducing the amount of privilege checks we need to make (we don't need an additional privilege check to redact object namespaces). This is an implementation detail that is not visible to consumers.

For collectMultiNamespaceReferences, if the user is not authorized to access a requested object, they will encounter a 403 error. If they are not authorized to access a referenced object, that will be silently filtered out of the results. This is consistent with our existing authorization checks for exporting saved objects with references. See unit tests and API integration tests for specific examples.

2. Removed outdated SOR/SOC methods:

I removed the addToNamespaces and deleteFromNamespaces methods, these are superseded by the new updateObjectsSpaces method. I updated existing client code in the "spaces" and "ml" plugins to use the new method.

3. Updated Copy-to-Space behavior

The copySavedObjectsToSpaces API has been improved for usage with multi-namespace types. Previously, if you attempted to copy an object to another space where it already exists, it would attempt to do so and return a conflict error. Then you would be able to resolve the conflict error by overwriting the same object with itself. While this technically worked, it was a confusing and poor user experience. Now if you attempt to copy object(s) to space(s) where they already exist, those will be skipped.

Note: we made a conscious decision not to do the same for resolveCopySavedObjectsToSpacesConflicts, as it's a lot of extra testing surface for less benefit. Technically one user could attempt a copy, then another user could share one of the affected objects to a destination space before the first user gets a chance to resolve the conflicts -- in this case, if the user had chosen to overwrite the destination object, it would be harmless and nothing would actually change.

What is not included

These will be addressed in a follow-on PR:

  • Client-side changes:
    • Update ShareToSpaceFlyout to prevent users from sharing an object into a space where it has a matching alias -- this situation would cause future calls to resolve to result in an alias conflict scenario. Instead, at share time, the user must decide to skip the changes, or to disable the legacy URL aliases in the destination space(s).
    • Update ShareToSpaceFlyout to fetch outbound references, and to display references/tags, and give users the option to exclude tags from the share operation.
    • Update the "Delete" modal on the Saved Objects Management page to warn users if they are attempting to delete an object that exists in multiple spaces (it would result in a 400 error), and allow them to do it anyway using the force option
    • Update the client-side SOC implementation to support the resolve method -- this should have been done in Phase 2.5 but it was missed because it does not rely on the same type definitions as the server-side SOC
  • API docs for /api/spaces/_get_shareable_references and /api/spaces/_update_objects_spaces in the new docs system
  • API docs for plugin authors to provide guidance on converting single-namespace object types into multi-namespace object types
  • Additional usage data collection for the resolve method (to track how many times legacy URL aliases are used in practice, and to track how many legacy URL aliases are disabled)

Testing

To manually test collectMultiNamespaceReferences functionality:

  • Install this plugin: https://github.com/jportner/kibana-sso-test-plugin
    It registers the toy saved object type (which is shareable), and allows you to easily generate new objects for testing.
  • Navigate to the plugin ("barney" in the nav pane) and click the "Get new toys" button to generate new objects.
  • Navigate to the Saved Objects Management page and use the Share action on any of the objects
  • "Sheriff Woody" is a special case, it has aliases in the "alpha", "bravo", and "charlie" spaces. If you look at HTTP responses in dev tools, when you open the Share flyout for this object, you will notice that the _get_shareable_references API returns a list of spaces where this object has conflicting aliases.
    • Note: you don't need to have actually created these three spaces for the purposes of this test -- but you can manually create each matching space (before or after) if you want. At the moment it will have no visible impact.

To manually test updateObjectsSpaces functionality:

  • Spin up Elasticsearch and Kibana with a trial license
  • Create additional spaces
  • Add sample data
  • Use the ML app to create jobs with the sample data
  • Use the ML job management page to share the jobs to other spaces

@jportner jportner added release_note:skip Skip the PR/issue when compiling release notes v7.13.0 v8.0.0 labels Mar 10, 2021
Copy link
Member

@legrego legrego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just did a high-level review of get_references_deep. I think this will satisfy our needs!

@jportner jportner force-pushed the sharing-saved-objects-phase-3 branch 3 times, most recently from bd111b5 to 2055343 Compare March 23, 2021 19:26
@legrego legrego self-requested a review March 26, 2021 14:58
@jportner jportner force-pushed the sharing-saved-objects-phase-3 branch from 2055343 to e85c529 Compare March 29, 2021 14:18
Copy link
Member

@legrego legrego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First real pass through - I haven't tested or reviewed any tests at this point. Nice work so far!

@jportner jportner force-pushed the sharing-saved-objects-phase-3 branch 2 times, most recently from 48121fc to e1ec80b Compare April 5, 2021 16:49
@jportner jportner added v7.14.0 and removed v7.13.0 labels Apr 7, 2021
@jportner jportner requested a review from watson April 8, 2021 14:28
@jportner jportner force-pushed the sharing-saved-objects-phase-3 branch from e1ec80b to 7f7b6f7 Compare April 8, 2021 22:23
@elastic elastic deleted a comment from kibanamachine Apr 8, 2021
@legrego legrego self-requested a review April 12, 2021 12:46
Copy link
Member

@legrego legrego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've only done some high-level smoke testing on this, but wanted to leave you with some comments for discussion before I go on PTO. Looking good so far!

Aside from these comments, I think I've gone as far as I'd like to go in terms of server-side code review before getting the core team's input, specifically on the inclusion of the tags SOC wrapper. Once we have consensus on that approach, then I'll do a final review with more in-depth testing.

Copy link
Contributor

@watson watson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finished reviewing collectMultiNamespaceReferences. Found nothing problematic - just a few nits and questions 😃

@jportner jportner force-pushed the sharing-saved-objects-phase-3 branch from 7f7b6f7 to 7862c95 Compare April 16, 2021 00:24
Copy link
Contributor

@pgayvallet pgayvallet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial review

Overall the implementation looks fine. A few nits and comments.

The only thing we really need to find the best solution for is the current relations/tags dissociation that complexify a lot the implementation and forced to create an additional SO wrapper just to work around the parameter conversion (#94383 (comment))

export interface SavedObjectsCollectMultiNamespaceReferencesOptions
extends SavedObjectsBaseOptions {
/** Whether or not to include tags when collecting references */
excludeTags?: boolean;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following the discussion started #67380 (comment),

If we were to have an excludeTags checkbox instead of the current includeTags in the UI, could we just get rid of this SavedObjectsCollectMultiNamespaceReferencesOptions.excludeTags property by having the UI just send typesToExclude: ['tag'] when the checkbox is checked?

I know this is poor isolation between the SOM ui and the tagging feature, but if that avoid adding this new SO client wrapper only to toggle an option, it seems like totally worth it.

Also, I agree that now that we no longer have an OSS distribution, we could have the tag SO type become more 'official', and maybe even put the tag concept/feature into the SOR, which would make the current workaround to just hardcode the typesToExclude value in the UI even more acceptable for the transition.

x-pack/plugins/security/server/saved_objects/index.ts Outdated Show resolved Hide resolved
x-pack/plugins/spaces/common/types.ts Outdated Show resolved Hide resolved
@pgayvallet
Copy link
Contributor

@jportner please ping me when you'll want a second pass :)

@jportner jportner force-pushed the sharing-saved-objects-phase-3 branch from 7862c95 to 092424e Compare May 3, 2021 01:10
@jportner jportner requested review from legrego and watson May 3, 2021 01:10
@jportner
Copy link
Contributor Author

jportner commented May 3, 2021

@legrego @watson this is ready for another round of reviews (I rebased onto master again and force-pushed -- see the 6 most recent commits for changes since your last reviews). I haven't addressed all of Pierre's feedback yet, but I modified the Secure SOC Wrapper implementation some more and added unit tests to address the deficiencies that I found.

Next I'm going to:

  • Add integration tests
  • Rip out the old addToNamespaces / deleteFromNamespaces APIs
  • Address the rest of PR review feedback
  • Work on client-side changes

Copy link
Member

@legrego legrego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job on the unit tests! The coverage is great, and they're fairly readable as far as SO tests go.

RE: client-side changes, would it make sense to split the remaining work into a different PR, so that you don't keep this one open for longer than necessary?


externalRouter.post(
{
path: '/api/spaces/_update_objects_spaces',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sure this is on your mental todo list, but for posterity: don't forget the API Docs 🙂

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didn't have any API Docs for the _share_saved_object_add and _share_saved_object_remove routes. Since we are moving away from Asciidocs, I plan to only make API Docs for the new _get_shareable_references and _update_objects_spaces routes using the new docs system. I'll add that in a follow-on PR along with other sharing-saved-objects-related guidance 👍

@jportner
Copy link
Contributor Author

jportner commented May 3, 2021

Great job on the unit tests! The coverage is great, and they're fairly readable as far as SO tests go.

Thanks!

RE: client-side changes, would it make sense to split the remaining work into a different PR, so that you don't keep this one open for longer than necessary?

I like that idea. The currently included client-side work should be enough to rip out addToNamespaces/deleteFromNamespaces.

@jportner jportner requested review from a team as code owners May 6, 2021 23:26
@jportner jportner requested review from legrego and pgayvallet May 6, 2021 23:26
Copy link
Member

@tsullivan tsullivan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addition to SavedObjectsCreatePointInTimeFinderOptions LGTM

@jgowdyelastic
Copy link
Member

ML changes tested. LGTM

Copy link
Contributor

@pgayvallet pgayvallet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looked at core's changes. Implementation did not change much.

LGTM, just a few remarks and questions

Comment on lines -19 to +23
hidden: true,
hidden: false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we now need this type to be visible?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we call collectMultiNamespaceReferences, for every object that we collect, we also check to see if it has any legacy URL aliases in any other spaces, and return that to the consumer via the spacesWithMatchingAliases field.

This search relies on the SOR's find method, so we can't do it if the object type is hidden, unless we want to add some sort of workaround. I didn't want to add anything else to the growing list of find options, so this seemed like the best approach.

* Note: if options.purpose is 'updateObjectsSpaces', it must be a shareable type (in other words, the object type must be registered with
* the `namespaceType: 'multi'`).
*/
export interface SavedObjectsCollectMultiNamespaceReferencesObject {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: add @public to all public types/interfaces of this file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I think I had asked him to remove @public 🙈. I was under the impression that it wasn't required -- anything that's eventually exported out of src/core/server should automatically be treated as "public", right? Is there another reason for including this tag?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@legrego if we don't include this tag, the top-level docs (src/core/public/public.api.md and src/core/server/server.api.md) include a warning before the interface, example:

// Warning: (ae-missing-release-tag) "SavedObjectsCollectMultiNamespaceReferencesObject" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
//
// @public
export interface SavedObjectsCollectMultiNamespaceReferencesObject {
...

Sorry I missed this, @pgayvallet! I'll add the @public tags back in 👍

Comment on lines +618 to +621
async collectMultiNamespaceReferences(
objects: SavedObjectsCollectMultiNamespaceReferencesObject[],
options?: SavedObjectsCollectMultiNamespaceReferencesOptions
): Promise<SavedObjectsCollectMultiNamespaceReferencesResponse> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR is not really introducing it, but we have an inconsistency on the space vs namespace naming. eg collectMultiNamespaceReferences versus updateObjectsSpaces

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it was my understanding that we want to eventually get rid of namespace terminology in favor of space now that we aren't shipping an OSS distribution. I agree this is confusing, especially because this is the first mention of "Spaces" in the Core codebase. I'm happy to change this to updateObjectsNamespaces if you think that would be better for now.

@lukeelmers lukeelmers self-requested a review May 10, 2021 15:05
Copy link
Member

@legrego legrego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, mostly nits and a couple of small items I think we should tweak as we prepare for The Merge™️

jportner added 2 commits May 12, 2021 12:53
Keeping the new unit tests, though. Modified them accordingly.
This was an oversight that I just noticed while updating TSdocs.
Copy link
Member

@lukeelmers lukeelmers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest changes to SO code LGTM! 🚀

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
spaces 259 258 -1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
ml 5.9MB 5.9MB -241.0B
spaces 287.2KB 288.1KB +943.0B
total +702.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
spaces 41.9KB 41.5KB -338.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id before after diff
legacy-url-alias - 4 +4
Unknown metric groups

API count

id before after diff
core 2182 2205 +23

References to deprecated APIs

id before after diff
canvas 29 25 -4
crossClusterReplication 8 6 -2
fleet 4 2 -2
globalSearch 4 2 -2
indexManagement 12 7 -5
infra 5 3 -2
licensing 18 15 -3
monitoring 109 56 -53
total -73

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jportner jportner requested a review from legrego May 14, 2021 13:46
Copy link
Member

@legrego legrego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great work as usual @jportner! 🎉

@jportner jportner added the auto-backport Deprecated - use backport:version if exact versions are needed label May 14, 2021
@jportner jportner merged commit b2d36b8 into elastic:master May 14, 2021
@jportner jportner deleted the sharing-saved-objects-phase-3 branch May 14, 2021 18:46
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request May 14, 2021
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
7.x

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request May 14, 2021
yctercero pushed a commit to yctercero/kibana that referenced this pull request May 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes v7.14.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sharing saved-objects in multiple spaces: phase 3
8 participants